Automatic vs. symbolic differentiation

[Dustin Webb]

Does Theano use automatic differentiation, symbolic differentiation, or some combination of the two? It is unclear from the documentation. Some parts even seem to conflate the ideas such as the statement "see automatic differentiation for a description of symbolic differentiation" in http://deeplearning.net/software/theano/tutorial/gradients.html.

Thanks,

--

Dustin Webb
Algorithmic Robotics Lab

Laboratoire d'Informatique des Systemes Adaptatifs

Defining Intelligence

[Olivier Delalleau]

It's automatic differentiation, at least according to a quick glance at http://en.wikipedia.org/wiki/Automatic_differentiation

-=- Olivier

--

---
You received this message because you are subscribed to the Google Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to theano-users...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Thomas Wiecki]

I thought it was symbolic differentation (as this page suggests http://deeplearning.net/software/theano/library/gradient.html)?

It's difficult to find good information on this but from what I could learn automatic differentation seems faster in most cases so I'm wondering if Theano could support AD as well (if it's not already doing that).

--

Thomas Wiecki

PhD candidate, Brown University

Quantitative Researcher, Quantopian Inc, Boston

[Jeremiah Lowin]

I think it is a semantic difference -- Theano implements "automatic differentiation" but refers to its own types as "symbolic" tensors (as opposed to "numeric" arrays), and so Theano's gradient implements an operation that Theano calls "symbolic differentiation". 

But the truth is a little ambiguous. Broadly speaking, "symbolic differentiation" refers to the manipulation of mathematical statements, and a symbolic differentiator knows the rules for taking the derivative of each element of a statement, or each basic operation. "Automatic differentiation" refers to the manipulation of a computer program, and an automatic differentiator knows the rules for taking the derivative of each element of a program, or each basic function call or control flow statement. These two things are not the same, though they are sometimes very similar. Symbolic differentiation requires the repeated application of differentiation operators, and often yields very complex derivatives. Reverse-mode automatic differentiation, on the other hand, replaces a chain of functions with a chain of derivative functions, and so the result is usually much more manageable. That's how Theano works: it builds up a computational chain, and then passes derivatives backwards through the chain's gradient functions (think backprop).

However, for very simple statements, they may work the same, which is why identifying the differences can be confusing. For control flows like "if" statements and loops, they could be wildly different -- AD will usually give far more efficient answers by taking advantage of its knowledge of the program structure.

[Frédéric Bastien]

Thanks for this great answer. Can I put this in Theano faq? This questions raise from time to time.

Also, one of the 2 methos is faster but use more memory. The other is slower but use less memory. Theano use the one that is the faster, but that use more memory.

Fred